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How  TRAs  Got  Started 


“Program  managers’  ability  to  reject  immature  technologies  is 
hampered  by  (1)  untradable  requirements  that  force  acceptance 
of  technologies  despite  their  immaturity  and  (2)  reliance  on 
tools  that  fail  to  alert  the  managers  of  the  high  risks  that  would 
prompt  such  a  rejection.”  GAO/NSI AD-99-1 62 

“Identify  each  case  in  which  a  major  defense  acquisition  program  entered 
system  development  and  demonstration  ...  into  which  key  technology  has 
been  incorporated  that  does  not  meet  the  technology  maturity  requirement ... 
and  provide  a  justification  for  why  such  key  technology  was  incorporated  and 
identify  any  determination  of  technological  maturity  with  which  the  Deputy 
Under  Secretary  of  Defense  for  Science  and  Technology  did  not  concur  and 
explain  how  the  issue  has  been  resolved.”  National  Defense  Authorization 
Act  for  Fiscal  Year  2002 

“The  management  and  mitigation  of  technology  risk,  which  allows  less  costly 
and  less  time-consuming  systems  development,  is  a  crucial  part  of  overall 
program  management  and  is  especially  relevant  to  meeting  cost  and  schedule 
goals.  Objective  assessment  of  technology  maturity  and  risk  shall  be  a  routine 
aspect  of  DoD  acquisition.”  DoDI  5000.2,  paragraph  3.7. 2. 2 


Stop  launching  programs  before  technologies  are  mature 


What  is  a  TRA? 


Systematic,  metrics-based 
process  that  assesses  the 
maturity  of  Critical 
Technology  Elements  (CTEs) 

-  Uses  Technology  Readiness 
Levels  (TRLs)  as  the  metric 

Regulatory  information 
requirement  for  all 
acquisition  programs 

-  Submitted  to  DUSD(S&T)  for 
ACAT  ID  and  1AM  programs 


*  Not  a  risk  assessment 

+  Not  a  design  review 

+  Does  not  address  system 
integration 


Critical  Technology  Element  (CTE)  Defined 


A  technology  element  is  “critical”  if  the  system 
being  acquired  depends  on  this  technology 
element  to  meet  operational  requirements  with 
acceptable  development  cost  and  schedule  and 

with  acceptable  production  and  operation  costs 
and  if  the  technology  element  or  its  application  is 

either  new  or  novel. 

Said  another  way,  an  element  that  is  new  or  novel  or 
being  used  in  a  new  or  novel  way  is  critical  if  it  is 
necessary  to  achieve  the  successful  development 
of  a  system,  its  acquisition  or  its  operational  utility. 

CTEs  may  be  hardware,  software,  manufacturing,  or  life  cycle  related 

at  the  subsystem  or  component  level 


Why  is  a  TRA  Important?  (i  of  2) 


The  Milestone  Decision  Authority 
(MDA)  uses  the  information  to  support 
a  decision  to  initiate  a  program 

-  Trying  to  apply  immature  technologies 
has  led  to  technical,  schedule,  and  cost 
problems  during  systems  acquisition 

-  TRA  established  as  a  control  to  ensure 
that  critical  technologies  are  mature, 
based  on  what  has  been  accomplished 


Congressional  interest 

-  MDA  must  certify  to  Congress  that 
the  technology  in  programs  has 
been  demonstrated  in  a  relevant 
environment  at  program  initiation 

-  MDA  must  justify  any  waivers  for 
national  security  to  Congress 
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Quantifying  the  Effects  of  Immature 

Technologies  <i  of 2) 


According  to  a  2005  GAO  review  of  54  DoD  programs: 

-  Only  15%  of  programs  began  SDD  with  mature 
technology  (TRL  7) 

•  Programs  that  started  with  mature  technologies  averaged 
9%  development  cost  growth,  a  7  month  schedule  delay, 
and  a  1%  acquisition  unit  cost  growth 

•  Programs  that  did  not  have  mature  technologies  averaged 
41%  development  cost  growth,  a  13  month  schedule  delay, 
and  a  21%  acquisition  unit  cost  growth 

-  At  critical  design  review,  42%  of  programs  demonstrated  design 
stability  (90%  drawings  releasable) 

•  Design  stability  not  achievable  with  immature  technologies 

•  Programs  with  stable  designs  at  CDR  averaged  6%  development  cost 
growth 

•  Programs  without  stable  designs  at  CDR  averaged  46%  development 
cost  growth  and  a  29  month  schedule  delay 


Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-05-301 ,  March  2005 
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Quantifying  the  Effects  of  Immature 

Technologies  (2  of 2> 


According  to  a  2006  GAO  review  of  52  DoD  programs: 

-  Only  10%  of  programs  began  SDD  with  mature 
technology  (TRL  7) 

•  Programs  that  started  with  mature  technologies  averaged 
4.8%  development  cost  growth  and  a  1%  acquisition  unit 
cost  growth 

•  Programs  that  did  not  have  mature  technologies  averaged 
34.9%  development  cost  growth  and  a  27%  acquisition  unit 
cost  growth 

-  Only  23%  of  programs  began  SDD  with  DoD’s  standard 
for  mature  technology  (TRL  6) 

•  Programs  that  started  with  mature  technologies  averaged 
18.8%  development  cost  growth 

•  Programs  that  started  with  mature  technologies  averaged 
34.6%  development  cost  growth 


Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-06-391 ,  March  2006 
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Why  is  a  TRA  Important?  (2  of  2) 


The  PM  uses  the  expertise  of  the  assessment  team  and 
the  rigor  and  discipline  of  the  process  to  allow  for: 

-  Early,  in  depth  review  of  the  conceptual  product  baseline 

-  Periodic  in-depth  reviews  of  maturation  events  documented 
as  verification  criteria  in  an  associated  CTE  maturation  plan 

-  Highlighting  (and  in  some  cases  discovering)  critical 
technologies  and  other  potential  technology  risk  areas  that 
require  management  attention  (and  possibly  additional 
resources) 

The  PM,  PEO,  and  CAE  use  the  results  of  the 
assessment  to: 

-  Optimize  the  acquisition  strategy  and  thereby 
increase  the  probability  of  a  successful  outcome 

-  Determine  capabilities  to  be  developed  in  the  next 
increment 

-  Focus  technology  investment 
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Joint  Capabilities  Integration  and  Development 

System  (JCIDS) 


Strategic  Guidance  - 

National  Security  Strategy/National  Defense  Strategy/National  Military  Strategy 


Family  of  Joint  Future  Concepts 
Concepts  of  Operations 
Joint  Tasks 


Integrated  Architectures 


Functional  Solution  Analysis 


Defense  Acquisition  System  -  DoD  5000 


Approach  1 


_ ^  Post 

Approach  2  ^  Independent 
^  Analysis 


Approach  N 


Initial 

Capability 

1 

Capability 

Capabilities 

Development 

Production 

Document 

Document 

Document 

(ICD) 

—  (CDD) 

_ ,  (CPD) 

\1  1 

Nl 

M 

^■1 

ll 


JCIDS  governed  by  --  CJCSI  3170 
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Overview  of  Technology  Considerations 
During  Systems  Acquisition 


Joint  Capabilities 
Integration  & 
Development 
System  (JCIDS) 


Process  entry  at  Milestones  A,  B,  or  C 

Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


(Program 

Initiation) 


IOC 


FOC 


Concept 

Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

\  Concept 
/  Decision 

Design 

<  y  Readiness 

V  Review 

LRIP/IOT&E  /S  Decision 

V  Review 

Pre-Systems  Acquisition 
ICD  CCD 


Systems  Acquisition 
CPD 


Sustainment 


TRAs  required  at  MS  B,  MS  C,  and  program 
initiation  for  ships  (usually  MS  A). 
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Technology  Considerations  Pre  Milestone  A 
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Technology  Considerations  During  the 
Technology  Development  Phase 


14 


Technology  Considerations  During  the  System 
Development  and  Demonstration  Phase 
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PM  responsibility 


Process  Overview 


Collect 

data 


PM  responsibility 
Coordinate  with  SAT  Exec 
Keep  DUSD(SAT)  informed 

PM  responsibility 
Best  Practice:  Independent 
review  team  appointed  by  SAT 
Exec  verifies 

PM  responsibility 
Coordinate  with  SAT  Exec 
Keep  DUSD(SAT)  informed 

SAT  Exec  responsibility 
Appoints  independent  review 
team  to  do  it;  PM  funds  it 


Coordinate  and  submit  TRA 


j 


SAT  Exec  coordinates 
Acquisition  Executive  submits 


/  \ 

OSD  review 

v _ ) 


DUSD(SAT)  responsibility 
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Set  schedule 


(  Responsible;  \ 

Vlnd  Review  Teanr\^ 

Collect 

x^Verifies^/ 

lUcMUiyiiiy  u  i  to 

. J . , 

Identify  CTEs 


X 


Coordinate  CTEs 


TRA  Schedule  Established 
CTE  Identification  Process 
Data  Collection 
CTEs  Coordinated 


Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week 
26  25  24  23  22  21  20  19  18  17  16  15 

i 


Schedule  should  be  set  6-12  months  before  the  Milestone  Review 
depending  on  the  complexity  of  the  program. 
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PM 

Responsible; 
vlnd  Review  Team 


CTE  Identification: 


verifies^/  Management  Process 

Initial  review 


-  PM  led  with  program  office,  system  contractors,  and 
Govt  labs 

-  Thorough,  disciplined  and  conservative  approach 

-  Identifies  longer  list  of  candidates  to  ensure  that  no 
potential  CTE  is  overlooked 

-  Identifies  information  needed  to  determine  whether 
the  candidates  meet  the  criteria  in  the  CTE  definition 

Independent  review 

-  Conducted  by  independent  review 
team  of  experts 

-  Resolves  status  based  on  data  and 
expertise 

-  Makes  recommendations  whether 
candidates  meet  the  criteria 


PM 

Responsible; 
vlnd  Review  Team 


CTE  Identification: 


VeTlfles  Technical  Process  (i  of  3) 


*  Utilize  the  work 
breakdown 


Aircraft  System 

I 


structure  (WBS) 
or  system 
architecture  for 
IT  systems,  to 
identify  CTE 
candidates  by: 


Sys  Engineering/ 
PgmManagement 
I  Peculiar 
|  Support  Equipmet  t 
Common 

Support  Equipmer  t 
|  Operational/ 
SiteActivitation 

Initial  Spares  & 
Repair  Parts 


Air  Vehicle  (AV) 


System  T&E 


S  Airframe  S  DT&E 

S  Propulsion  S  OT&E 

S  AV  Sys  Software  S  Mock-ups 
S  Comm/ID  S  T&E  Support 

S  Central  Computer  S  Test  Facilities 
S  Fire  Control 
S  Auto  Flight  Control 
S  Weapons  Delivery 


Training 


S  Equipment 
S  Services 
S  Facilities 


S  Tech  Pubs  S  Construction/ 

S  Eng  Data  Conversion/ 

S  Mgt  Data  Expansion 

S  Support  Data  S  Equipment 

S  Data  Depository  Acquisition  or 
Modernization 
S  Maintenance 
QndusFacilitie! 

Adapted  from  MIL-HDBK-881 


-  Establishing  the  functions  to  be  performed  by  each  system, 
subsystem,  or  component  throughout  the  WBS 


-  Determining  how  the  functions  will  be  accomplished 

-  Identifying  the  technologies  needed  to  perform  those 
functions  at  the  desired  level 
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At  least  one  answer 
must  be  "yes" 


PM 

Responsible; 
vlnd  Review  Team 


CTE  Identification: 


VeTlfles  Technical  Process  (2  of  3) 


*  Criticality  to  the  program  criteria 

-  Does  the  technology  directly  impact 
an  operational  requirement? 

-  Does  the  technology  have  a 
significant  impact  on  an  improved 
delivery  schedule? 

-  Does  the  technology  have  a 
significant  impact  on  the 
affordability  of  the  system? 


See  section  D.4  of  the  Deskbook  for  other  examples 


21 


At  least  one  answer 
must  be  "yes" 


PM 

Responsible; 
vlnd  Review  Team 


CTE  Identification: 


VeTlfles  Technical  Process  (3  of  3) 


•  New  or  novel  criteria 

-  Is  the  technology  new  or  novel? 

-  Is  the  technology  modified? 

-  Has  the  technology  been 
repackaged  such  that  a  new 
relevant  environment  is  realized? 

-  Is  the  technology  expected  to 
operate  in  an  environment  and/or 
achieve  a  performance  beyond  its 
original  design  intention  or 
demonstrated  capability? 


Environment  key  to  “new  or  novel” 
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Environment  Examples 


ftjrb-orne  Target 


Physical  Environment,  for  instance  Mechanical  Ci 
Processors,  Servers  and  Electronics;  Kinetic  and  Kinematic; 
Thermal  and  Heat  Transfer;  Electrical  and  Electromagnetic; 
Climatic-Weather,  Temperature,  Particulate;  Network 
Infrastructure 

Logical  Environment,  fgr  instance,  Software  (Algorithm) 
Interfaces;  Security  Interfaces;  Wejb-enablement 

Data  Environment,  for  instance,  Data  Formats  and  Databases; 
Anticipated  Data  Rates,  Data  Delay  and  Data  Throughput;  and 
Data  Packaging  and  Framing 

Security  Environment,  for  instance, 'Connection  to  Firewalls; 

r  Ur  burr*  Targe  1 

Security  Appliques;  Rates  and  Methods  of  Attack  Ground  Target 

User  and  Use^HVironment,  for  instance,  Scalability; 
Upgradeability;  User  ^p^vior  Adjustments;  User  Interfaces; 
Organization  Change/ReaTignments  with  System  Impacts; 
Implementation  Plan _ ^ 


Others  may  be  relevant 
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Sample  Questions  to  Determine  if 
Environment  is  New  or  Novel 


•  Is  the  physical/logical/data  environment  in  which  this 
CTE  has  been  demonstrated  similar  to  the  intended 
environment?  How  is  it  different?  Is  the  difference 
important? 

•  Is  the  CTE  going  to  be  operating  at  or  outside  of  the 
usual  performance  envelope?  Do  specifications 
address  the  behavior  of  the  CTE  under  these 
conditions?  What  is  unique  or  different  about  the 
proposed  operations  environment? 

•  Do  test  data,  reports  or  analysis  that  compare  the 
demonstrated  environment  to  the  intended  environment 
exist?  If  modeling  and  simulation  is  an  important  aspect 
of  that  comparison,  are  the  analysis  techniques 
common  and  generally  accepted? 


See  Section  D.3.2  of  the  Deskbook  for  more  questions 
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CTE  Coordination  and 
Data  Collection 


PM  submits  list  to  the  Component  S&T  Executive 
and  requests  a  TRA 


S&T  Executive  may  add  CTEs  if  it  is  felt  that  special 
attention  is  warranted 

PM  collects  evidence  of  CTE  maturity 

-  Ongoing  process  throughout  CTE  identification 

-  May  include  component  and  subsystem  test  . 
descriptions,  analyses,  environments,  and 
results 

-  Best  Practice:  evidence  should  be  as  objective 
as  possible  and  align  with  current  technology 
maturation  plan's  documented  verification 
criteria  for  achieving  the  next  level 


Keep  DUSD(S&T)  informed;  may  suggest  additional  CTEs 


Component 
SAT  Exec 
Responsible 


Assessing  CTE 
Readiness 


Assess  CTEs;  prepare  TRA 


I 


Coordinate  and  submit  TRA 


OSD  review 


Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week 
14  13  12  11  10  9  8  7  6  5  4  3  2  1 


TRA  Performed 
TRA  Coordination 

DUSD(S&T)  TRA  Review  &  Evaluation 

Independent  TRA  (if  necessary) 

Evaluation  Memo 

Milestone  Review 


1 
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TRL  Overview 


*  Measures  technology  maturity 

*  Indicates  what  has  been  accomplished  in  the 
development  of  a  technology 

-  Theory,  laboratory,  field 

-  Relevant  environment,  operational 
environment 

-  Subscale,  full  scale 

-  Breadboard,  brassboard,  prototype 

-  Reduced  performance,  full 
performance 

*  Does  not  indicate  that  the  technology  is  right  for 
the  job  or  that  application  of  the  technology  will 
result  in  successful  development  of  the  system 
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Increasing  maturity 


Hardware  and  Manufacturing  TRLs 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 
9. 


Basic  principles  observed  and  reported 


Technology  concept  and/or  application 
formulated 

Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept 

Component  and/or  breadboard 
validation  in  a  laboratory  environment 

Component  and/or  breadboard 
validation  in  a  relevant  environment 

System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 


System  prototype  demonstration  in  an 
operational  environment 

Actual  system  completed  and  qualified 
through  test  and  demonstration 

Actual  system  proven  through 
successful  mission  operations 


Increasing  maturity 


Software  TRLs 


1.  Basic  principles  observed  and  reported. 

2.  Technology  concept  and/or  application  formulated. 

3.  Analytical  and  experimental  critical  function  and/or 

characteristic  proof  of  concept  Q 

4.  Module  and/or  subsystem  validation  in  a  50 

laboratory  environment,  i.e.  software  prototype 
development  environment 

5.  Module  and/or  subsystem  validation  in  a  relevant 
environment 

6.  Module  and/or  subsystem  validation  in  a  relevant 
end-to-end  environment 

7.  System  prototype  demonstration  in  an  operational 
high  fidelity  environment 

8.  Actual  system  completed  and  mission  qualified 
through  test  and  demonstration  in  an  operational 
environment 

9.  Actual  system  proven  through  successful  mission 
proven  operational  capabilities 
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TRA  Performed 


*  Program  responsible  for  funding,  BUT 
most  of  the  work  has  already  been  done 

*  Independent  team  trained  and  convened 
by  the  Component  S&T  Executive  to 

-  Make  the  assessments 

-  Write  the  report 

*  Multiple  TRAs  should  be  conducted  if 
multiple  systems  still  in  competition 


Contact  DUSD(S&T)  with  any  issues  (e.g.,  CTE  uncertainty) 

early  in  the  process 


See  additional  hardware  examples 
in  Section  C.2  of  the  Deskbook 


See  additional  software  examples 
in  Section  C.3  of  the  Deskbook 


See  additional  manufacturing  examples 
in  Section  C.4  of  the  Deskbook 
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Component  TRA  Coordination 


Identified  CTEs  and  assessed  TRL 


-  Component  TRA  approval  is  an  agreement  on  its  accuracy  only 
Maturity  requirements: 

-  Subsystem  demonstrated  in  relevant  environment  (TRL  6)  for  MS  B 

-  Prototype  (TRL  7)  or  actual  system  for  manufacturing  CTEs  (TRL  8) 
demonstrated  in  an  operational  environment  for  MS  C 


Three  options  if  a  technology  is  not  mature 

-  Request  a  delay  for  the  Milestone  review  until  all 
CTEs  are  at  the  requisite  maturity  level 

-  Utilize  alternative,  mature  technologies 

-  As  a  last  resort,  carry  immature  technologies  *■'/  ■  )  llfl  / ill 

into  the  Milestone  review  and  prepare  a  waiver, i  * 

based  on  inability  to  meet  national  security  \, 
objectives,  for  the  MDA  to  submit  to  Congress  , ^  * 
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DUSD(S&T)  TRA  Review 


Results  of  initial  review 

-  Concur 

-  Request  revisions 
Results  of  final  review 

-  Concur 

-  Concur  with  reservations 

-  Perform  independent  technical  assessment 


DUSD(S&T)  informs  Milestone  Decision  Authority  of  the  results 
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Outline 


•  Introduction 

-  What  is  a  TRA 

-  Why  it  is  important 

•  Overview  of  Technology  Considerations  During 
Systems  Acquisition 

•  Synopsis  of  the  TRA  Process 

-  Identifying  Critical  Technology  Elements  (CTEs) 

-  Assessing  CTE  Readiness 

•  Pre-Milestone  B  Technology  Maturation  Policy 

•  How  the  S&T  Community  Can  Best  Support  the  TRA 
Process 

-  Do’s  and  Don’ts 

•  References  and  Resources 
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Technology  Maturation  Policy  Leading  to 
Milestone  B  is  Unambiguous 


“The  project  shall  exit  Technology  Development  when 
an  affordable  increment  of  militarily-useful  capability 
has  been  identified,  the  technology  for  that  increment 
has  been  demonstrated  in  a  relevant  environment,  and  a 
system  can  be  developed  for  production  within  a  short 
timeframe  (normally  less  than  five  years);  or  when  the 

MDA  decides  to  terminate  the  effort . A  Milestone  B 

decision  follows  the  completion  of  Technology 
Development.”  (DoDI  5000.2,  paragraph  3.6.7) 


Technology  Development 


Technology  Maturation  Policy  Leading  to 
Milestone  B  is  Unambiguous  (cont’d) 


“The  management  and 
mitigation  of  technology  risk, 
which  allows  less  costly  and 
less  time-consuming  systems 
development,  is  a  crucial  part  of 
overall  program  management 
and  is  especially  relevant  to 
meeting  cost  and  schedule 
goals. 


Objective  assessment  of  technology 
maturity  and  risk  shall  be  a  routine  aspect 
of  DoD  acquisition.  Technology  developed 
in  S&T  or  procured  from  industry  or  other 
sources  shall  have  been  demonstrated  in  a 
relevant  environment  or,  preferably,  in  an 
operational  environment  to  be  considered 
mature  enough  to  use  for  product 
development  in  systems  integration. 
Technology  readiness  assessments,  and 
where  necessary,  independent 
assessments,  shall  be  conducted. 

If  technology  is  not  mature,  the 
DoD  Component  shall  use 
alternative  technology  that  is 
mature  and  that  can  meet  the 
user’s  needs.”  (DoDI  5000.2, 
paragraph  3. 7. 2.2) 


and 
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The  Policy  is  Reflected  as  a  Title  10 
Requirement  for  Certification 


10  USC  §2366a  states 

Major  defense  acquisition  programs:  certification 
required  before  Milestone  B  or  Key  Decision  Point 
B  approval: 

(a)  CERTIFICATION.  A  major  defense  acquisition 
program  may  not  receive  Milestone  B  approval , 
or  Key  Decision  Point  B  approval  in  the  case  of 
a  space  program ,  until  the  milestone  decision 
authority  certifies  that  - 

(2)  the  technology  in  the  program  has  been 

demonstrated  in  a  relevant  environment; 

•  •  • 

(10)  the  program  complies  with  all  relevant  policies, 
regulations  and  directives  of  the  Department  of 
Defense 


'THE  UNDER  SECRETARY  OF  DEFENSE 
3010  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-3010 


MAYO  2  2006 


MEMORANDUM  FOR  SEE  DISTRIBUTION 


SUBJECT :  Implementation  of  Section  2366a  of  Title  10.  Unites  States  Code 


Seaton  2366a  of  title  10.  United  States  Code,  as  enacted  by  section  SOI  of  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2006  (Pub  L.  No.  109-163),  requires  the  Milestone 
Decision  Authority  (MDA)  for  a  Major  Defense  Acquisition  Program  (MDAP)  to  maie 
certain  certifications  prior  to  Milestone  B  or  Key  Decision  Point  B  approval 

To  fulfill  this  requirement,  the  MDA.  without  the  authority  to  delegate,  shall  sign  a 
memorandum,  subject  'Program  Certification. '  prior  to  sigmng  the  Acquisition  Decision 
Memorandum  (ADM).  This  certification  memorandum  shall  be  prepared  'for  the  record." 
and  shall  include  the  statements  in  the  attachment,  without  modification.  If  the  program  is 
initiated  at  a  later  decision  point  e  g..  Milestone  C.  a  similar  memorandum  shall  be 
prepare!  as  a  matter  of  policy,  consistent  with  the  mrenr  of  the  statute  The  certification 
memorandum  shall  be  submitted  to  the  congressional  defense  committees,  as  defined  at  10  U 
S.C  101_(16).  with  the  first  Selected  Acquisition  Report  for  the  program  after  completion  of 
the  certification 

The  MDA  may  waive  one  oc  more  of  components  (1)  through  (6)  of  the  required 
certification  (specifically,  one  or  more  of  paragraphs  (1)  through  (6)  in  the  attachment)  for  an 
MDAP  if  the  MDA  detetmmes  that  hut  for  such  a  waiver,  the  Department  would  be  unable 
to  meet  critical  national  security  objectives.  The  MBA  shall  submit  the  waiver,  the 
determination,  and  reasons  for  the  determination,  in  writing,  to  die  congressional  defense 
committees  within  30  days  of  authorizing  the  waiver.  The  MDA  may  not  delegate  this 
waiver  authority 

In  addition  to  die  certification  memorandum,  the  MDA  will  include  die  following 
statement  in  die  ADM:  '!  base  renewed  the  program  and  have  made  the  certifications 
required,  or  executed  a  waiver  as  authorized,  by  section  2366a  of  title  10.  United  States 
Code/* 

This  policy  shall  apply  to  MDAPs  approved  by  me  and  to  MDAPs  managed  by 
Department  of  Defense  Component  Acquisition  Executives  or  the  Assistant  Secretary 
of  Defense  for  Networks  and  Information  Integration.  This  requirement  went  into  effect 
January  6. 20C6.  and  shall  be  reflected  m  die  next  revision  to  Department  of  Defense 
In-r-tticn  C  1 


Attachment 
As  stated 


Certification  Submitted  with  the  First  Selected 
Acquisition  Report  for  the  Program 
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•  •  •  But  Waivers  Are  Allowed 


(c)  WAIVER  FOR  NATIONAL  SECURITY.  The 
milestone  decision  authority  may  waive  the 
applicability  to  a  major  defense  acquisition 
program  of  one  or  more  components  (as  specified 
in  paragraph  ( 1 ),  (2),  (3),  (4),  (5),  (6),  (7),  (8),  or  (9)  of 
subsection  (a))  of  the  certification  requirement  if 
the  milestone  decision  authority  determines  that , 
but  for  such  a  waiver ,  the  Department  would  be 
unable  to  meet  critical  national  security  objectives. 

Whenever  the  milestone  decision  authority  makes  such  a 
determination  and  authorizes  such  a  waiver ,  the  waiver ,  the 
determination ,  and  the  reasons  for  the  determination  shall  be 
submitted  in  writing  to  the  congressional  defense  committees 
within  30  days  after  the  waiver  is  authorized. 
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Changes  Made  to  Meet  the  Statutory 

Requirements 


USD(AT&L)  policy  determinations 

-  Programs  will  no  longer  be  initiated 
with  immature  technologies 

-  The  same  standards  apply  to  all 
acquisition  programs 

DDR&E  will  provide  technical  advice 
in  support  of  certification 

-  TRA  will  be  the  basis  of  that  advice 
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Outcomes  from  Process  Changes  in 
Support  of  the  New  Situation 


Safeguards  in  place  to  provide  the  DDR&E 
with  the  confidence  necessary  to  assure  the 
MDA  that  certification  can  be  made 

-  To  make  the  TRA  support  the  certification,  it 
must  draw  upon  the  best  technical  information 
available  prior  to  source  selection 

Assurance  that  technologies  have  been 
demonstrated  in  a  relevant  environment  by 
the  winning  SDD  Phase  contractor 

-  To  initiate  programs  with  mature  technologies, 
the  source  selection  process  should  include  a 
focus  on  technical  maturity 


Outline 


•  Introduction 

-  What  is  a  TRA 

-  Why  it  is  important 

•  Overview  of  Technology  Considerations  During 
Systems  Acquisition 

•  Synopsis  of  the  TRA  Process 

-  Identifying  Critical  Technology  Elements  (CTEs) 

-  Assessing  CTE  Readiness 

•  Pre-Milestone  B  Technology  Maturation  Policy 

•  How  the  S&T  Community  Can  Best  Support  the  TRA 
Process 

-  Do’s  and  Don’ts _ 

•  References  and  Resources 
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Use  of  TRA  and  TRL  Terminology 

Acquisition  Community 


•  TRAs  help  ensure  the  technology  being  used 
in  acquisition  programs  is  mature 

-  Use  of  immature  technologies  leads  to  cost 
growth  and  schedule  slippage 

•  TRAs  provide  the  basis  for  DDR&E  to  advise  the  MDA 
on  10  USC  2366a  certification  for  technology  maturity 
at  Milestone/KDP  B 

•  TRLs  are  the  maturity  metric  for  CTEs  in  TRAs;  TRLs 
5-9  are  applicable 

-  Environments  and  the  performance  requirements 
defined  by  a  program  of  record 

•  TRAs  performed  at  Milestones  B  and  C  and  program 
initiation  for  ships 
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Use  of  TRA  and  TRL  Terminology 

S&T  Community 


•  TRAs  are  an  acquisition  construct 

-  They  are  not  performed  on  an  S&T  project 

•  TRLs  may  be  used  as  a  maturity  metric  for 
technologies  in  a  technology  development 
project;  TRLs  1-6  are  applicable 

-  TRL  definitions  and  descriptions  successively 
spell  out  progress  (as  measured  by  tests)  toward  a 
goal 

•  TRLs  may  be  used  as  part  of  a  technology 
managers’  ongoing  assessment  of  a  technology 
or  technologies 


Use  of  TRLs  5  and  6  overlaps  with  the  acquisition 

community 
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Issues  Arising  from  Overlap  in  Terminology 


•  Normally  demonstrating  a  technology  beyond  TRL  4 
requires  more  resources  than  maturing  the  technology 
through  TRLs  1-3 

-  Higher  level  assemblies  needed 

-  More  refined  components  needed 

-  More  broad  scale  tests  needed 

•  Such  resources  often  obtained  from  programs  of  record  as 
activities  shift  from  the  realm  of  technology  advancement 
to  technology  transition  and  insertion 

•  Misunderstanding  of  TRLs  5-6  has  led  to  misuse  of  the 
terminology  when  competing  for  these  resources 

-  May  damage  the  S&T  program  and/or  the  TRA  process 

-  May  create  the  wrong  impression  with  leadership 

Misuse  of  TRA  and  TRL  terminology  and  concepts 
may  lead  to  negative  unintended  consequences 
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TRL  4  is  the  Breakpoint  between  Invention 

and  Application  <i  of  3> 


TRLs  1-3  involve  development  of 
functionality,  mostly  independent  of  the 
application 

To  achieve  TRL  4 

-  Must  begin  integration  of  components  to 
represent  how  they  would  be  used  in  a 
fieldable  application 

-  Must  have  a  generic  application  in  mind 
without  knowing  exactly  how  that 
application  will  be  used 
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TRL  4  is  the  Breakpoint  between  Invention 

and  Application  <2of3) 


•  To  achieve  TRL  5  or  higher 


-  Must  be  in  the  context  of  an  application 
for  a  program  of  record 

•  The  application  provides  the  both  the 
metric  (speed,  energy  density...)  and  the 
threshold  (10  m/s;  100J/g...) 

-  Must  have  an  understanding  of  the 
relevant  environment 


•  The  relevant  environment  cannot  be 
determined  without  an  understanding  of 
requirements  and  intended  operational 
use  defined  in  programmatic  documents 
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TRL  4  is  the  Breakpoint  between  Invention 

and  Application  <3of3) 


•  Gun  propellant  example 

-  TRL  1:  Theoretical  studies  and  computer  models 
lead  to  synthesis  and  characterization  of  a  new 
energetic  material  for  a  propellant 

-  TRL  2:  New  material  synthesized,  characterized 
and  potential  performance  of  propellants  mapped 
via  computer  codes 

-  TRL  3:  New  propellants  prepared  at  small  scale 

-  and  performance,  processing,  and  physical 
properties  characterized 

-  TRL  4:  Based  on  TRL  1-3  data  propellant  designed 
for  a  specific  application  and  near  full  scale  tests 
performed  to  confirm  computer  modeling 

-  TRL  5:  New  propellant  produced  in  quantity  and 
evaluated  in  near-final  system  configuration 
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MS  B  Requirement:  Demonstration  or  Validation 
in  a  Relevant  Environment  (TRL  6) 


Relevant  Environment:  a  set  of 
stressing  conditions 
representative  of  the  full  spectrum 
of  relevant  operational 
employments,  which  are  applied  to 
a  CTE  as  part  of  a  component  (TRL 
5)  or  system/subsystem  (TRL  6) 
model  or  prototype  in  order  to 
identify  whether  any  design 
changes  or  fixes  are  needed  to 
support  the  required  (threshold) 
functionality 
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Demonstration  or  Validation  of  a  Technology  in 
a  Relevant  or  Operational  Environment 


*  Requires  successful  trial  testing  that 
either: 

-  Shows  that  the  technology  satisfies 
functional  need  across  the  full  spectrum 
of  operational  employments,  or 

-  Shows  that  the  technology  satisfies  the 
functional  need  for  some  important 
(stressing)  operational  employment  and 
uses  accepted  techniques  to  extend 
confidence  over  all  required  operational 
employments 
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S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  <i  of  7) 


•  Not  labeling  the  technology 
assessments  performed  by  the  S&T 
community  as  a  TRA 

-  Misuses  the  term  in  a  way  that 
misleads  stakeholders 

-  May  damage  both  TRA  and  technology 
proponent’s  reputation 

•  Not  justifying  the  need  for  research 
(dollars)  based  on  achieving  TRL  5/6 
without  the  metrics  and  the  threshold 
provided  by  a  program  of  record 
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S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  <2of7) 


•  Applying  judgment  when  determining  the  relevant 
environment  from  the  operational  environment  to 
maximize  test  efficiency 

-  Environments  tested  should  be  stressing  enough  to  be 
persuasive 

•  Being  exhaustive  is  usually  too  expensive 

Example 

•  Launching  a  satellite  should  not  be  on  critical  path 
for  design  and  demonstration 

Relevant  environment  depends  on  what  is  stressing 

-  E.g.,  thermal  load,  radiation  in  space,  g-forces  during 
launch 

-  May  be  tested  and  demonstrated  in  the  lab 

•  Technical  expertise  ensures  stressing  portion  of  the 
environment  is  demonstrated;  no  expensive, 
exhaustive  tests  applied  to  non-critical  element 
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S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  <3of7) 


•  Continuing  promising  technology  development  at  TRL 
4  when  there  is  no  program  of  record 

-  While  TRLs  5/6  are  achieved  with  a  successful 
demonstration,  there  are  a  large  number  of  useful 
activities  that  could  take  place  at  TRL  4 

•  It  is  helpful  to  complete  an  extensive  performance 
characterization  rather  than  a  “point  demonstration” 

-  Provides  information  on  how  to  incorporate  the  technology 
into  a  design 

-  Enables  more  rapid  insertion 

-  Supports  knowledge-based  acquisition  decisions 

•  A  technology’s  capability  may  be  advanced  using  metrics 
of  interest  without  knowing  the  particular  thresholds 

•  Improvements  may  be  planned  on  the  basis  on  draft 
requirements 

Continuing  development  applies  to  TRL  5/6  as  well 
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S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  (40f7> 

•  Preparing  to  help  programs  of  record  achieve  TRLs  5/6  via  expertise 
with  the  technology  itself  and  test  design  as  they  reach  back  to  the 
tech  base  for  solutions 

-  Neither  labs  nor  program  offices  are  organized  or  staffed  to  conduct  the 
realistic  demonstrations  of  highly  integrated  components  need  to  mature 
technologies  to  TRL  5/6  on  their  own 

-  S&T  personnel  (and  institutions)  should  transition  into  a  supporting  role 

Example 

Armor  piercing,  fin  stabilized  discarding  sabot  (APFSDS) 
had  nearly  maxed  performance  capabilities 

Armament  Enhancement  Initiative  in  1984  established  to 
reduce  sabot  weight  (partnership  between  S&T  and 
acquisition) 

By  1987,  requirement  established  for  new  composite 
sabot;  fielded  in  1992 

Cycle  repeated  itself  for  fielding  more  advanced  sabot  in 
2003 
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S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  <5of7) 


•  Differentiating  proof  of  principle  demonstration 
(TRL  3)  from  demonstration  in  a  (requirements 
defined)  relevant  environment  TRL  5/6 

-  For  TRL  3,  do  not  need  to  have  an  application  in 
mind 

*  Acquisition  customer  may  say  “If  you  make  it  work, 

I’ll  use  it” 

-  For  TRL  5,  there  must  be  an  application  and 
components  must  be  representative  of  use  in 
intended  application 

-  For  TRL  6,  ready  to  turn  it  over  to  a  design 
engineer 

Overselling  technology  readiness  damages  the 

S&T  community  credibility  as  much  as  overselling 
technology  performance.  May  lead  to  •  •  • 


S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  <6of7) 


•  •  •  acquisition  problems  if  program  initiated 
with  immature  technology 


Example 

Regenerative  Liquid  Propellant  Gun  ultimately 

became  the  CRUSADER  program 

-  When  program  transitioned  from  S&T,  the  concept 
was  proven 

•  All  technology  issues  were  reasonably  well  recognized 

•  Plan  was  to  solve  the  problems  in  engineering 

-  Eventual  failure  (program  cancellation)  was 
associated  with  the  difficulties  when  transferring  the 
technology  to  practical  hardware 
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S&T  Practices  to  Better  Support  TRAs  and 
the  Acquisition  Process  <7  of 7> 


•  Avoiding  the  use  of  TRLs  as  a  sole  and  governing 
measure  for  managing  S&T  programs 


-  TRLs  are  a  static  metric;  represent  snapshot  in  time;  they 
do  not  assess  difficulty  of  advancement 

-  TRLs  lack  high  specificity;  much  more  information  needs  to 
be  conveyed 

•  Should  lay  out  specific  technical  goals  to  evaluate  technology 
status  /  progress 

-  Could  lead  to  a  premature  stoppage  of  development  efforts 
as  soon  next  TRL  is  reached 

Using  TRLs  a  high-level  metric  for  managing  a  balanced 
portfolio  of  investments  from  basic  research  to 
exploratory  development  of  components 


-  Helps  avoid  under  emphasis  on  basic  principles  or  concept 
formulation  (TRLs  1  and  2)  in  favor  of  research  on  proof  of 
principle  or  demonstration  in  lab  (TRLs  3  and  4) 


Acquisition  Practices  to  Improve  Linkages 

with  S&T  (i  of  2) 


Developing  (in  conjunction  with  the  S&T  community)  a 
technology  maturation  plan  to  identify  how  technologies  will 
be  demonstrated  in  a  relevant  environment  by  Milestone  B 


Establishing  measurable  technical  performance 
requirements  as  technology  transition  exit  criteria  to  achieve 


TRL  6  for  CTEs 

-  Fully  describe  the 
relevant  environment  in 
technology  transition 
agreements 

-  Include  metrics  and 
thresholds  in  a  relevant 
environment 

-  Do  not  specify  TRL  6  as 
an  exit  criterion 


Acquisition  Practices  to  Improve  Linkages 

with  S&T  <2  of  2) 


•  Shifting  necessary  resources  (funding  and  personnel) 
to  the  technology  development  phase 

•  Accounting  for  the  event-driven  nature  of  S&T 
processes  when  developing  schedules 


Applying  schedule-driven  constraints  may  compromise 
technology  development  and  lead  to  immature 
technologies  at  Milestone  B 

Backup  plans  and  alternatives  to  technologies  less  than 
TRL  6  are  advisable 
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Outline 


*  Introduction 
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Systems  Acquisition 
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